An On-Line Time Warping Algorithm for Tracking Musical Performances

Dynamic time warping is not suitable for on-line applications because it requires complete knowledge of both series before the alignment of the first elements can be computed. We present a novel online time warping algorithm which has linear time and space costs, and performs incremental alignment of two series as one is received in real time. This algorithm is applied to the alignment of audio signals in order to track musical performances. 1 On-Line Time Warping Although efficiency and real-time concerns of dynamic time warping (DTW) have been addressed in the literature, we do not know of any work in which the real-time constraint involves a streamed sequence, so that the alignment must be calculated incrementally, in the forward direction, while one of the sequences is not known in entirety. In this work we present an on-line time warping (OLTW) algorithm which is able to perform incremental alignment of arbitrarily long sequences in real time. DTW aligns time series U = u1, ..., um and V = v1, ..., vn by finding a minimum cost path W = W1, ...,Wl, where each Wk is an ordered pair (ik, jk), such that (i, j) ∈ W means that the points ui and vj are aligned. The alignment is assessed with respect to a local cost function d(i, j), usually represented as an m × n matrix, which assigns a match cost for aligning each pair (ui, vj). The path cost is the sum of the local match costs along the path. Several constraints are placed on W , namely that the path is bounded by the ends of both sequences, and it is monotonic and continuous. The minimum cost path can be calculated in quadratic time by dynamic programming, using the recursion: D(i, j) = d(i, j) + min { D(i, j − 1) D(i− 1, j) D(i− 1, j − 1) } where D(i, j) is the cost of the minimum cost path from (1, 1) to (i, j), and D(1, 1) = d(1, 1). The path itself is obtained by tracing the recursion backwards from D(m,n). In the on-line case, the length of the incoming sequence is unknown, so one of the boundary conditions for the search ∗This work was supported by the Vienna Science and Technology Fund project CI010 Interfaces to Music and the EU-FP6-IST507142 project SIMAC. ÖFAI acknowledges the financial support of the Austrian federal ministries BMBWK and BMVIT. must be estimated along with the optimal path. Also, in order to run in real time with arbitrarily long series, the complete algorithm must be linear in the length of the series, so that the incremental step is bounded by a constant. If U is the partially unknown sequence, then we seek at each time t the best alignment of the sequence u1, ..., ut to some initial subsequence of V . This is performed with the OLTW algorithm (Figure 1), which we now explain. The variables t and j are pointers to the current positions in series U and V respectively. The main loop of the algorithm calculates a partial row or column of the path cost matrix. The calculation of the path cost uses the standard DTW recursion formula, restricted to use only the matrix entries which have already been calculated. The path cost is normalised by the path length, so that paths of varying lengths can be compared in the function GetInc. The number of cells calculated is given by the search width parameter, c; e.g. for a new row, the row number is incremented, and the cells in the last c columns up to and including the current column are calculated. The function GetInc selects whether to calculate a row, column, or both. If less than c elements of each series have been processed, new rows and columns are alternately calculated. If one sequence has been incremented successively MaxRunCount times, the other sequence is incremented. Otherwise the minimum path cost for each cell in the current row and column is found. If this occurs in the current position (t, j), then both the row and column counts are incremented; if it occurs elsewhere in row j, then the row count is incremented, otherwise the column count t is incremented. For each incoming data point ut, the minimum cost path calculated at time t is the same as that calculated by DTW, assuming the same path constraints, but the number of calculations performed by OLTW is bounded by a constant. A further advantage of OLTW is that the centre of the search band is adaptively adjusted to follow the best match. 2 Tracking of Musical Performances In music performance, high level information such as structure and emotion is communicated by the performer through parameters such as tempo, dynamics, articulation and vibrato. These parameters vary within a musical piece, between musical pieces and between performers. We use OLTW to extract this information directly from audio signals by aligning different performances of the same piece of music, enabling live tracking and visualisation of expressive parameters during a performance. This could be used to complement the listening ALGORITHM On-Line Time Warping t := 1; j := 1 previous := None INPUT u(t) EvaluatePathCost(t,j) LOOP IF GetInc(t,j) != Column t := t + 1 INPUT u(t) FOR k := j c + 1 TO j IF k > 0 EvaluatePathCost(t,k) IF GetInc(t,j) != Row j := j + 1 FOR k := t c + 1 TO t IF k > 0 EvaluatePathCost(k,j) IF GetInc(t,j) == previous runCount := runCount + 1 ELSE